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PART I: Summary of Required Elements for State Accountability 
Systems 

Summary of Implementation Status for Required Elements of 
State Accountability Systems 



Principle 1 : All Schools 



1 .1 Accountability system includes all schools and districts In the state. 

1 .2 Accountability system holds all schools to the same criteria. 

1 .3 Accountability system incorporates the academic achievement standards. 

1 .4 Accountability system provides Information In a timely manner. 

1 .5 Accountability system includes report cards. 

1 .6 Accountability system includes rewards and sanctions. 



Principle 2: All Students 



2.1 The accountability system includes all students 

2.2 The accountability system has a consistent definition of full academic year. 

2.3 The accountability system properly includes mobile students. 



Principle 3: Method of AYP Determinations 



F 



F 



F 



F 



F 



3.1 

3.2 



3.2a 

3.2b 

3.2c 



Accountability system expects all student subgroups, public schools, and LEAs to reach 
proficiency by 201 3-14. 

Accountability system has a method for determining whether student subgroups, public 
schools, and LEAs made adequate yearly progress. 

Accountability system establishes a starting point. 

Accountability system establishes statewide annual measurable objectives. 
Accountability system establishes intermediate goals. 



Principle 4: Annual Decisions 




The accountability system determines annually the progress of schools and districts. 



STATUS Legend: 

F - Final state policy 

P - Proposed policy, awaiting State approval 
W - Working to formulate policy 
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Principle 5: Subgroup Accountability 



F 



F 



F 



F 



F 



F 



5.1 

5.2 

5.3 

5.4 

5.5 

5.6 



The accountability system includes all the required student subgroups. 

The accountability system holds schools and LEAs accountable for the progress of student 
subgroups. 

The accountability system includes students with disabilities. 



The accountability system includes limited English proficient students. 

The State has determined the minimum number of students sufficient to yield statistically 
reliable information for each purpose for which disaggregated data are used. 

The State has strategies to protect the privacy of individual students in reporting 
achievement results and in determining whether schools and LEAs are making adequate 
yearly progress on the basis of disaggregated subgroups. 



Principle 6: Based on Academic Assessments 




Accountability system is based primarily on academic assessments. 



Principle 7: Additional Indicators 

F 7.1 Accountability system includes graduation rate for high schools. 

P -j ^ Accountability system includes an additional academic indicator for elementary and middle 

schools. 



F 7.3 Additional indicators are valid and reliable. 



Principle 8: Separate Decisions for Readinq/Lanquaqe Arts and Mathematics 

P g ^ Accountability system holds students, schools and districts separately accountable for 
reading/language arts and mathematics. 



Principle 9: System Validity and Reliability 



F 



F 



F 



9.1 

9.2 

9.3 



Accountability system produces reliable decisions. 

Accountability system produces valid decisions. 

State has a plan for addressing changes in assessment and student population. 



Principle 10: Participation Rate 

P .|Q Accountability system has a means for calculating the rate of participation in the statewide 
assessment. 

F 10 2 Accountability system has a means for applying the 95% assessment criteria to student 
subgroups and small schools. 

STATUS Legend: 

F - Final policy 

P - Proposed Policy, awaiting State approval 
W- Working to formulate policy 
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PART II: State Response and Activities for Meeting State Accountability System 
Requirements 

Principle 1 : All Schools 

1.1 Accountability system includes all schools and district in the state. 

Utah requires all public schools and LEAs, including those serving special populations 
and charter schools, to participate in the Utah Performance Assessment System for 
Students (U-PASS). Special population schools include the Utah Schools for the Deaf 
and the Blind, and Youth in Custody (YIC). Several key steps in the accountability 
process assure inclusion of all students, schools, and LEAs. 

• Eor many years, the Utah State Office of Education (USOE) has pre-printed 
assessment answer sheets for all Utah schools. This pre-printing is based on 
official state IDs and school names. At the time of USOE test scoring, the 
USOE is able to assure that all schools are participating in testing. 

• In the past year, the USOE has refined the system to include a second stage of 
verification. School names and IDs are checked against the official state list 
that is maintained by the USOE School Einance and Statistics Section. 

• With the implementation of the Utah Alternate Assessment program (UAA) 
those schools that serve students who have severe disabilities are able to fully 
participate in testing and accountability. 

• Einally, USOE, in collaboration with LEAs, can ensure that YIC students also 
participate in testing and accountability. 

U-PASS includes course/grade specific criterion-referenced tests (Core CRTs) as well as 
norm-referenced tests, performance tasks and diagnostic testing. Eor compliance with 
federal No Child Left Behind (NCLB) legislation, the Core CRTs from U-PASS are used 
to report academic achievement. Using a subset of tests from U-PASS allows Utah to 
utilize all rules, auditing procedures, and practices already in place to meet NCLB 
requirements. 
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1.2 Accountability system holds all schools to the same eriteria. 

U-PASS holds all sehools aeeountable to the same eriteria as mandated by the Utah 
Legislature. Central eomponents of state and federal aceountability are the state’s high 
quality standards-based eriterion-refereneed assessments (Core CRTs). Utah has a long 
history of standards-based assessments aligned to the Utah Core Currieulum, whieh is a 
eomprehensive eurrieulum for eaeh subjeet at eaeh grade-level/eourse as mandated by 
state law. The first use of Utah’s Core CRTs was in the 1989-1990 sehool year. 

Performanee standards have been established for all Core CRTs. Aeross all tested grades 
and subjeet areas, the Bookmark Technique was utilized. To assure appropriate use of 
the method, USOE staff eonsulted with original authors of the teehnique. The same 
performanee standards are approved for the Utah Alternate Assessment (UAA). 
Establishing the same performanee standards allows the state to inelude speeial 
population sehools that do not take the Core CRTs in standard or modified eonditions, 
but do partieipate in the UAA. 

Other non-testing indieators (graduation rate and attendanee) are uniformly defined and 
implemented in sehools and EEAs across the state. This uniformity of measures assures 
that the same eriteria are applied in determining EEA and sehool AYP status. 
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1.3 Accountability system incorporates, at a minimum, a definition of basic, proficient, 
and advanced student aehievement levels in reading/language arts and mathematies. 

Utah has defined profieieney levels for all Core CRTs. The eut seores for eaeh Core 
CRT were established using the Bookmark Teehnique. Four levels are defined: Level 1: 
Minimal, Level 2: Partial, Level 3: Suffieient, Level 4: Substantial. The Utah State Board 
of Edueation adopted the new titles and deseriptors on April 4, 2003. The deseriptors for 
eaeh level as well as the mateh to federal profieieney levels are outlined in Table 1. 



Table 1: Utah Student Aehievement Level Matehed to Federal Levels 



Level 


Deseriptor 


Federal Fevels 


Level 4: 
Substantial 


A student seoring at this level is profieient on 
measured standards and objeetives of the Core 
Currieulum in this subjeet. The student's 
performanee indieates substantial understanding and 
applieation of key eurrieulum eoneepts. 


Advaneed 


Level 3: 
Suffieient 


A student seoring at this level is profieient on the 
measured standards and objeetives of the Core 
Currieulum in this subjeet. The student's 
performanee indieates suffieient understanding and 
applieation of key eurrieulum eoneepts. 


Profieient 


Level 2: 
Partial 


A student seoring at this level is not yet profieient on 
measured standards and objeetives of the Core 
Currieulum in this subjeet. The student's performanee 
indieates partial understanding and applieation of key 
eurrieulum eoneepts. 


Basie 


Level 1: 
Minimal 


A student seoring at this level is not yet profieient on 
measured standards and objeetives of the Core 
Currieulum in this subjeet. The student's performanee 
indieates minimal understanding and applieation of 
key eurrieulum eoneepts. 
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1 .4 Accountability system provides information in a timely manner. 

AYP information will be available to the districts for a 30 day review on August 15. 
After the 30 day review, the information will be released to the public. See section 1.7 
for a description of the application of sanctions based on this timeline. 

Table 2 outlines the data elements/events, purposes, and date due for the 2004-2005 
school year and all years thereafter. All dates refer to the time by which the required data 
are to be “clean and final,” but not the date of final submission. 



Table 2: Data Submission Timeline 



Data Element/Event 


Purpose 


Date 


Test Pre-print file 


Pre-print of test answer documents 


April 1 


Test Window 


Eanguage Arts test completion 


Eive weeks, beginning 
five weeks prior to last 
Monday of school year 


Mathematics test completion 


Eive weeks, beginning 
five weeks prior to last 
Monday of school year 


Test Answer 
Documents 


Scoring of student answer 
documents by USOE 


East day of district school 
calendar (1 week after 
close of test window) 


Raw Data Return 


Preliminary Data Review by EEA 


4 days after submission of 
answer document 


End of Year 
Clearinghouse Data 


Student level demographic data 
including cumulative add/drop 
codes 


July 15 


Begin of Year 
Clearinghouse Data 


Drop-out data 


October 15 



For many years, the USOE has scanned and scored Core CRTs. This has been a great 
benefit for quality control and scoring turnaround. The testing window for the Core 
CRTs maximizes the days of instruction but still facilitates timely scoring and reporting 
of results. 

Raw scores will be distributed to LEAs for review and preliminary analysis. This 
preliminary analysis allows EEAs to project AYP results and anticipate budget and 
notification actions. See section 1.7 for details. 

The USOE developed and implemented a state Data Warehouse. The Data Warehouse 
serves as the foundation for the analysis and generation of school and district 
performance reports and determination of EEA and school AYP prior to the start of a 
new school year. Appeals will be allowed at the school and EEA level. 

The timeline currently in place allows schools enough time to notify parents about public 
school choice or supplemental educational service options. 
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Student Pre-Print Data (March/April) - Preprint student data is submitted to USOE. 
This data includes student demographic data. At the district and school level, this data is 
summarized and sent to districts to approve or make corrections in their operational 
student data and then resubmit it. This individual information is used to pre-print Core 
CRT answer sheets. This process both reduces work for the students and improves 
student data. Districts provide written approval of their pre-print data before answer 
sheets are printed. 

Core CRT Scanning & Scoring (May/June) - Math and Language Arts Core CRTs are 
scanned and scored by USOE. Utah is one of only a few states that do in-house scanning 
and scoring. This approach leads to faster scoring turnaround, stronger quality assurance 
procedures and more accurate test data transfer to our USOE Data Warehouse. 
Throughout the scanning process, inconsistencies and problems with student answer 
sheets are identified and corrected. Depending on the issue, either the district or school is 
directly involved. USOE scoring staff is assigned certain districts. This has enabled a 
relationship to develop with the districts and increases the cooperation as scanning or 
scoring issues arise. Scoring turnaround and testing procedures are well documented in 
Utah Board rule, R277-473-3. Time Periods for Administering and Returning Test 
Materials. 

Clearinghouse “End of Year” data submission (July) - Prom an accountability 
perspective, this student level data submission provides background characteristics, grade 
level data, enrollment date, exit date, graduation status, and attendance. This serves as the 
foundation of student level identification to which test data is matched. These data are 
passed on to the USOE Data Warehouse for integration with the other data systems. 

Teacher and Course Data (i.e., CACTUS) submission (July) - Teacher and course 
level data is maintained in the CACTUS database. This database is centralized by USOE. 
CACTUS serves as the operational teacher system and is updated throughout the school 
year. Districts are required to have all changes into the system finalized by July 

Behavior and Assessment Data Merge (July) - USOE data warehouse merges various 
data systems needed for AYP calculations. Utah has common naming and id system that 
districts have employed. 

Appeals and Consequences to missing data deadlines - Board Rule R277-484 . Data 
Standards, Deadlines and Procedures gives clear and specific detail regarding data 
deadlines. Districts must meet data deadlines or funds will be withheld. This rule also 
specifies procedures to get an extension for a deadline. 

Supporting evidence and background: 

Utah State Board Rule R277-484 
http://www.usoe.kl2.ut.us/data/R277-484.pdf 
U-PASS/NCLB Data Memo - Appendix H 
External Audit Report 2002 

http://www.usoe.kl2.ut.us/admin/projects/mgt/prelim%20report.pdf 
Testing & Scoring Procedures - Appendix C 



8 




1.5 Accountability system includes report eards for publie sehools and LEAs. 



AYP reports for sehools and LEAs serves as the report eard, inelude aggregated and 
disaggregated aeademie aehievement data by raee/ethnieity, poverty, and students with 
limited English profieieney, students reeeiving speeial edueation serviees, gender, and 
migrant status. Eor determining AYP, only total sehool, raee/ethnieity, poverty, and 
students with limited English profieieney, students reeeiving speeial education services 
are used. 

Distriets are required by law to make these reports available to the publie either in print 
or eleetronie eopy. In addition to the distriet level distribution, reports are available 
through USOE. 
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1.6 Accountability system includes rewards and sanetions for pub lie sehools and LEAs. 

Utah eomplies with NCLB requirements to identify for improvement, take eorreetive 
aetion, and restrueture sehools based on AYP for sehools reeeiving Title I funds. In 
addition, LEAs and sehools are identified for exemplary performanee. All publie sehools’ 
status with regard to AYP is made available to the publie through performanee reports 
and media sourees. 

The LEA whieh is identified for improvement, will exit improvement when it does not 
miss AYP in the same subjeet and in all grade spans for two eonseeutive years, or it does 
not miss AYP for the other aeademie indieator in all grade spans for two eonseeutive 
years. 



Utah LEA and State Sanctions 


Eor Title I sehools failing to make AYP, the LEA must, in the subsequent sehool year, do 
the following: 


Sehool 
Year 1 


Warning issued. Superintendents are eneouraged to take immediate aetion to 
assist toward improvement. 


Year 2 
SPI 1 


• LEA designates sehool as needing improvement before the beginning 
of the sehool year following failure to make AYP. 

• LEA must notify all parents of sehool’ s designation. 

• Sehool develops or revises a two-year aehievement plan to be 
approved by LEA within three months of improvement designation. 
Plan must inelude required elements: seientifieally-based researeh, 
sueeessful aetions, professional development ineluding 10% of its 
Title I Part A funds, speeifie annual, measurable goals and targets, 
fundamental teaehing and learning needs, parental involvement, 
additional time for learning, and shared responsibility for 
improvement. 

• LEA and/or qualified others provide teehnieal assistanee* with 
seientifieally based quality support. 

• LEA eonvenes peer review proeess within 45 days of reeeiving the 
plan or sooner, as possible. 

• Sehool implements plan expeditiously but not later than next full 
sehool year. 

• State ensures that LEA has assisted well. 

• State provides assessment data. 

• LEA provides ehoiee option and transportation to all students to 
another publie/eharter sehool not under improvement, as desired by 
parent/student. 


*Teehnieal assistanee must inelude at least six elements: Seientifieally based researeh; 
analysis of data; parental involvement; sustained, aligned, and foeused professional 
development; instruetional strategies; and foeused budget and resourees. 



Year 3 • Designation 

SPI 2 • Notifieation 
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• Technical assistance 

• Choice, where feasible 

• EEA arranges provision for supplemental service to all eligible 
students. EEA provides annual notice of eligibility for supplemental 
services. 

• State assigns monitor 


Year 4 
SPI3 


• Designation. 

• Notification 

• Technical assistance 

• Choice 

• Supplemental services 

• Required corrective action and notice of corrective action. 

• State continues monitoring. 


Year 5 
SPI4 


• Designation. 

• Notification 

• Technical assistance 

• Choice 

• Supplemental services 

• State approves or revises corrective action and advises Utah State 
Board of Education of designation. 

• EEA develops plan for alternative governance provisions 

• State intervention at EEA level possible 


Year 6 
SPI5 


• Designation 

• Notification 

• Technical assistance 

• Choice 

• Supplemental services 

• Corrective action 

• EEA must implement plan for alternative governance provisions 

• State intervention at EEA level possible 



There are no sanctions or designations for non-Title I schools beyond those specified in 
U-PASS for all Utah schools. U-PASS specifies school improvement plans for all Utah 
schools, regardless of Title I status. 

Rewards 

Schools making AYP and those making significant improvement are recognized by the 
USOE. All schools and LEAs are acknowledged, regardless of Title I status. Recognition 
may include: media/press release; visits from dignitaries; certificates, banners, or plaques. 

Supporting evidence and background: 

U-PASS Eegislation & State code regarding testing - Appendix A. 
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PRINCIPLE 2. All Students 



2.1 Accountability system includes all students. 

By law, all Utah students participate in the Core CRT program. Over the past three 
years, because of state and federal accountability, several steps have been refined to 
ensure inclusion of all students. Many of these modifications were implemented to meet 
state accountability and federal special education legislation (i.e., IDEA 1997). 

For the last 10 years, the USOE has pre-printed answer sheets for Utah students. This 
pre-printing has increased the quality and quantity of student test data. Pre-print files are 
submitted to the USOE by EEAs and are compared with other student level data 
submitted by EEAs, including the S3 and clearinghouse data (see section E4 for details). 

Utah has defined appropriate accommodations for students with disabilities. The use of 
Core CRTs has allowed for a variety of accommodations to be selected by the lEP team 
and still yield valid results that do not affect the test score interpretation. 

For the past three years, Utah has implemented alternate assessments for students with 
the most significant cognitive disabilities. The purpose of the alternate assessment is to 
measure the achievement of students with disabilities against alternate academic 
achievement standards as defined by the State. The two assessment approaches provided 
in Utah are out-of-level testing and Utah’s Alternate Assessment (UAA). 

Utah’s Alternate Assessment (UAA) is designed for assessing students with severe 
disabilities, especially those with the most significant cognitive disabilities. Objectives 
for the UAA are based on the student’s lEP goals. Although the UAA is operational, it is 
in the final stages of test refinement. Changes currently underway will ensure accurate 
measurement of achievement against alternate academic achievement standards. For the 
2004-05 school year, alignment of the UAA objectives with the extended core curriculum 
(through early literacy and numeracy skills) will be finalized. 

Out-of-level tests are allowed in language arts and mathematics, but only for students 
enrolled in special education. lEP teams may specify the administration of a test that is 
below the grade of enrollment at a level corresponding with the student’s grade level of 
instruction. Results are reported in relation to the test level rather than the grade of 
enrollment. In determining AYP, out-of-level tests will be counted as “not proficient”, 
but will count as participating in testing. 

Youth in Custody (YIC) students, as well as students in special purpose schools, also 
participate in testing and accountability. YIC facilities work with EEAs to obtain testing 
materials. The EEAs must either declare the YIC program as a school or include them 
with other an existing school. 

Students in the United States for less than one year will not be required to take the 
English Eanguage Arts CRT or the Science CRT, but will be required to take the math 
CRT. CRT scores for FEE students in their first year in the United States will not be 
used for AYP calculations. All other FEE students will participate in accordance with 
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NCLB with appropriate accommodations. These accommodations may include, but are 
not limited to, linguistic accommodations and small group settings 
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2.2 Accountability system has a consistent definition of full aeademie year. 

The USOE defines a full aeademie year as one in whieh students are in membership, in 
the same sehool, on September 15 and for not less thanlbO days. This definition 
represents a proxy for eontinuous enrollment, whieh may be literally determined by the 
LEA during review of AYP results. 

Data neeessary for implementing Utah’s definition of full aeademie year have been 
eolleeted for many years through the elearinghouse file. Temporary non-attendanee due 
to suspensions does not eonstitute a break in enrollment. A student transferred to another 
sehool or LEA is eonsidered no longer eontinuously in membership at the original sehool 

Students in membership in more than one sehool within a single LEA on September 15 
and not less than 160 days will be ineluded in determining the AYP of the LEA. Einally, 
all Utah students who are in membership in a Utah sehool for a full aeademie year are 
part of the statewide AYP ealeulation, ineluding students who have not been in 
membership in any single LEA for the full aeademie year. 
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2.3 Accountability system properly ineludes mobile students. 



The development of the Data Warehouse allows the USOE to inelude and track students 
who move frequently. The Warehouse gathers data from a variety of eollection sources, 
ineluding test data, S3 data, and elearinghouse data (see section 1.4 for details). 

From a testing perspective, all students, regardless of the amount of time they have been 
at a sehool, partieipate in the Core CRTs at the end of the sehool year. The reporting of 
sehool and LEA Core CRT performanee results ineludes all students ineluding mobile 
students and other speeial populations, sueh as students reeeiving speeial edueation 
services and students with limited English proficiency. 



Supporting evidence and baekground: 

USOE Data Warehouse (information & background) 
http://www.usoe.kl2.ut.us/upass/ 

Warehouse Data Dietionary 

http://www.usoe.kl2.ut.us/upass/oet29meeting/Visio- 

warehouse%201oad.pdf 

U-PASS Legislation & State code regarding testing - Appendix A. 
Future Sehool Report - Appendix E 
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PRINCIPLE 3: Method of AYP Determinations 



3.1 Accountability system expects all student subgroups, public schools, and LEAs 
to reach proficiency by 2013-14. 

Utah’s accountability system uses three criteria for AYP determination: participation rate, 
academic achievement, and an additional indicator. These three criteria are applied to 
both mathematics and reading/language arts separately. 

Utah has incorporated the NCLB expectation that by 2013-14 all student subgroups, 
schools, and LEAs will demonstrate proficiency in the areas of reading/language arts and 
mathematics. Utah’s plan includes all public, charter, and special population schools. 

To determine if yearly progress is made, starting points have been set for mathematics 
and reading/language arts at each of two levels: elementary /middle school and high 
school. The starting point was set according to federal guidelines (see section 3.2a for 
details). Based on the starting point using 2001-2002 school year data and the ultimate 
goal of 100% of students proficient by 2013-2014 school year, intermediate goals are set 
based on the statewide annual measurable objectives (see sections 3.2b and 3.2c for 
details). Schools must meet the annual measurable objectives or make acceptable 
improvement (safe harbor provision) for academic achievement as part of the 
requirements for making AYP. 

The other two criteria - participation rate and additional indicators are also used to make 
AYP determinations for student subgroups, schools, and LEAs. Details of these two 
criteria are in sections 7 and 10 respectively. 



16 




3.2 Accountability system has a method for determining whether student subgroups, 
publie sehools, and LEAs made adequate yearly progress. 

Utah’s accountability plan uses three eriteria - participation rate, aeademic aehievement, 
and additional indieator - for determining in student subgroups, sehools, and LEAs have 
made adequate yearly progress. The Data Warehouse serves as the data souree for 
making the determination of AYP for student subgroups, sehools, and LEAs. As 
deseribed in seetion 1.4, the Data Warehouse stores data from test files, elearinghouse 
data (ineluding student demographie information), and S3 year-end data eontaining 
additional student information for determining membership in relevant subgroups. 

Academic Aehievement 

In determining AYP for student subgroups, sehools, LEAs and the state as a whole, Utah 
eompares the pereent of students profieient to the annual measurable objeetive, examines 
partieipation rates, and examines the additional aeademie indieator. 

A student subgroup, sehool, or LEA of 10 or more students must meet or exeeed the 
annual measurable objeetive for reading/language arts and mathematies, must meet the 
participation rate for subgroups of 40 students or more, and meet the state’s requirement 
for another aeademie indieator (attendance for elementary and middle sehools and 
graduation for high sehools). If a student subgroup, sehool, or LEA fails to meet or 
exeeed the annual measurable objeetive, it must have redueed the pereent of students not 
profieient by the appropriate pereentage (safe harbor). If the safe harbor provision is 
employed, the additional indieator must be disaggregated and used for determining AYP 
for the subgroup. 

Table 3 displays the impaet data for various N sizes on student subgroups. These data 
indieate that the minimum N seleeted by Utah, N=10, will allow the maximum number of 
sehools to be held aeeountable for student subgroup performanee. 



Table 3: Impact Data for Various Minimum N Sizes - Grades 3-8 



Number of Sehools with Subgroups Ineluded in AYP Analysis for Various Minimum N Sizes 

(Sehool with >10 students = 522) 




Minimum N 




10 


25 


30 


50 


75 


Sehool 


522 










White 


511 


486 


483 


471 


440 


Low Ineome 


491 


444 


427 


355 


265 


Students with Disabilities 


450 


280 


227 


63 


11 


Limited English Proficient 


308 


173 


149 


81 


41 


Hispanie 


294 


161 


144 


87 


48 


Blaek 


29 


2 


1 


0 


0 



In ealculating AYP for LEAs, sehools, and subgroups, Utah employs a test of statistical 
significance with a one-tailed alpha of 0.01 for academic achievement of the annual 
measurable objeetive. This allows sehools with small subgroup populations to be held 
aeeountable without falsely identifying a sehool. This ereates a balanee between validity 



17 





(holding schools accountable for all students) and reliability (assuring that those 
subgroups identified have not been so identified simply on the basis of random 
fluctuation of scores). For AYP determination based on the annual measurable objective, 
a test of statistical significance will be applied for subgroups 3 10. The null hypothesis is 
that the observed percent of students proficient in any subgroup is equal to the required 
percent proficient defined by the annual measurable objective. The test of statistical 
significance is a z-score with the distribution of school mean scores (in terms of percent 
correct) around the null hypothesis. A school or LEA makes AYP if the null hypothesis is 
not rejected. 

In calculating AYP, any LEA, school, or student subgroup that does not meet the annual 
measurable AYP goal, must decrease the percentage of students not proficient by at least 
10 percent in accordance with the safe harbor provision. Utah employs a test of 
statistical significance using a one-tailed alpha of 0.25 for determining school-wise safe 
harbor (improvement). 

Participation Rate 

Participation in Core CRTs will be determined for each student subgroup, school, and 
LEA by comparing the testing pre-print file, test answer documents, all student file and 
clearinghouse data using the Data Warehouse. This process is used for all student 
subgroups, schools, and LEAs. Eor details about participation rate, see section 10. 

Additional Indicators 

The additional indicators for Utah’s accountability plan are attendance 
(elementary /middle schools) and graduation (high schools). These indicators can be 
calculated using data in the Data Warehouse. Schools and LEAs must meet the state 
goals for each indicator to make AYP. In addition, a student subgroup must meet the 
additional indicator goals if the safe harbor provision is employed for the student 
subgroup. The state goals for the additional indicators are 93% attendance and 85.7% 
graduation rate. Eor details about these indicators, see section 7. 

Because Utah cannot currently determine graduation rate for all subgroups, a proxy 
additional indicator will be used to determine AYP for 2003-2007. This proxy will be 
attendance. The attendance rate will be obtained by dividing the sum of days in 
attendance by the sum of days in membership across all students in the school and LEA. 
The goal for attendance will be set at 93%, consistent with the additional indicator goal 
for elementary and middle schools (see section 7.2). Schools must meet this goal or 
improve from the previous year. 

Eor a school that meets the annual measurable objective for all subgroups and thus uses 
only school level data for the additional indicator, graduation will be used. However, if a 
school employs safe harbor for a subgroup, AYP will be determined using graduation for 
the total school and attendance for the student subgroup(s). This temporary situation will 
apply equal criteria to all student subgroups while still holding the school accountable for 
graduation. Beginning in 2008, graduation will be the only additional indicator used at 
the high school level. 
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3.2a Accountability system establishes a starting point. 



A baseline is separately established for reading/language arts and mathematics based on 
NCLB requirements using the 20th percentile school by enrollment ordered by percent 
proficient for each subject. A separate starting point is established for grade spans 3-8 
and grade 10. The same starting point and the same annual measurable objeetives apply 
to ah student subgroups within the grade spans, culminating in 100 percent proficiency of 
ah students by 2013-2014. 

Table 4 displays student performance by proficiency. Table 5 displays performance based 
on grade level for the Grade 3-8 grade span. Finally, Table 6 displays the 20* percentile 
score for Utah schools by enrollment, when ranked by performance. 



Table 4; Number and Percent of Students by Proficiency - Grades 3-8 



Overall 


Proficient 
(Level 3 and 4) 


Not Proficient 
(Levels 1 and 2) 


Mastery 

Missing 


Total 

Included 


Language Arts 


153,335 (73.74%) 


54,599 (26.26%) 


7,888 


207,934 


Mathematics 


140,303 (67.29%) 


68,202 (32.71%) 


7,317 


208,505 



Table 5: Performance by Grade - Grades 3-8 





Mathematics 


Language Arts 


Grade 


% 

Proficient 


%Not 

Proficient 


Total 

Included 


% 

Proficient 


%Not 

Proficient 


Total 

Included 


3 


71 


29 


35,498 


76 


24 


35,544 


4 


71 


29 


36,292 


77 


23 


36,169 


5 


71 


29 


35,319 


77 


23 


35,211 


6 


61 


39 


34,967 


71 


29 


35,396 


7 


65 


35 


33,249 


79 


21 


32,941 


8 


65 


35 


33,180 


62 


38 


32,673 



Table 6: Starting Point - ( 


Trades 3-8 and High School 


Content Area 


Percent Proficient 


Language Arts 3-8 


65% 


Mathematics 3-8 


57% 


Language Arts HS 


64% 


Mathematics HS 


35% 



To verify that the 20* percentile score by enrollment, once ranked by performanee, was 
higher than the lowest student subgroup performance, the student subgroup data were 
analyzed. Table 7 displays student subgroup performance results for the 3-8 grade span. 
These results verify the use of the 20* percentile scores for Utah’s baseline value. 
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Table 7; Student Subgroup Performance - Grades 3-8 





Language Arts 


Mathematics 


Subgroup 

(Ordered by number of students) 


% 

Proficient 


Total 

Included 


Percent 

Proficient 


Total 

Included 


Caucasian 


78% 


176,164 


71% 


175,985 


Low Income 


59% 


66,081 


53% 


66,707 


Students with Disabilities 


31% 


23,029 


28% 


23,770 


Limited English Proficient 


48% 


19,148 


43% 


19,754 


Hispanic 


46% 


18,881 


40% 


19,467 


American Indian 


42% 


3,481 


37% 


3,529 


Asian/Pacific Islander 


75% 


3,342 


74% 


3,392 


Black 


53% 


2,175 


42% 


2,189 



Table 8 displays the student subgroup performance on Language Arts for 10* grade in 
Utah. These data are included to verify the 20* percentile starting point, which is 64 
percent proficient. 



Table 8: 10* Grade Language Arts 



Student Subgroup 
(Ordered by number of students) 


% Proficient 


Total Included 


Caucasian 


76.6% 


25,868 


Low Income 


58.3% 


5,449 


Hispanic 


47.0% 


1,617 


Limited English 


46.0% 


1601 


Special Education 


22.4% 


1,573 


Asian 


68.9% 


437 


American Indian 


33.2% 


335 


Pacific Islander 


53.9% 


325 


Black 


51.9% 


212 


Total 


73.9% 


37,417 
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3.2b Accountability system establishes statewide annual measurable objectives. 

Utah has established separate annual measurable objectives for reading/language arts and 
mathematics. Annual measurable objectives use the same percent proficient as the most 
recent intermediate goal. The reading and mathematics annual measurable objectives 
apply to ah student subgroups, schools, LEAs, and the state as a whole. A separate 
starting point will be established for grades spans 3-8 and grade 10. When calculating the 
percent proficient for a school with multiple grade levels, as well as the subgroups within 
them, the annual measurable objective is an aggregate of the percent of proficient 
students at ah grade levels in the school (a weighted average). 
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3.2c Accountability system establishes intermediate goals. 



Utah incorporates intermediate goals for the minimum percentage of students achieving 
proficiency. 

The intermediate goals assume equal increases expected eaeh year based on the 
proficiency gap between the baseline percent proficient and 100 percent proficient 
divided by the 12 intervening years (annual increase). However, the required percent 
proficient is not increased each year, rather it increases every other year and in the final 
year of the program. Table 9 displays these values for Reading/Language Arts and 
Mathematics, based on the results of the 2002 administration of the Core CRTs. 



Spring 2002 
Spring 2005 
Spring 2007 
Spring 2009 
Spring 2011 
Spring 2013 
Spring 2014 



Starting Point 

Goal TStarting Point + (Annual increase * 2) 
Goal 2: Spring 2004 Goal + (Annual increase * 2) 

Goal 3: Spring 2006 Goal + (Annual increase * 2) 

Goal 4: Spring 2008 Goal + (Annual increase * 2) 

Goal 5: Spring 2010 Goal + (Annual increase * 2) 

Final Goal: 100 % proficient 



Table 9: Starting Point and Intermediate Goals 


- Grat 


les 3-8 and HS 




Starting 

Point 


Goal 

1 


Goal 

2 


Goal 

3 


Goal 

4 


Goal 

5 


Final 


Language Arts Grades 3-8 


65 % 


71 % 


77 % 


83 % 


89 % 


95 % 


100% 


Mathematics Grades 3-8 


57 % 


64% 


71 % 


78 % 


85 % 


92% 


100% 


Language HS Grade 10 


64% 


70% 


76% 


82% 


88 % 


94% 


100% 


Mathematics Grade 10-12 


35% 


47% 


59% 


72% 


84% 


96% 


100% 
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PRINCIPLE 4: Annual Decisions 



4.1 Accountability system determines annually the progress of schools and LEAs. 

AYP decisions are made annually for each LEA and school. AYP determination is based 
on percent of students proficient or the reduction of students not proficient, participation 
rate, and achievement of the additional academic indicator. Reading/language arts and 
mathematics are examined separately for each of the aforementioned criteria for AYP. 
Annual measurable objectives for academic achievement are based on the most recent 
intermediate goal. 
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PRINCIPLE 5: Subgroup Accountability 



5.1 The accountability system includes all the required student subgroups. 

Both state accountability and NCLB require the disaggregation and reporting of the 
following student subgroups: 

Race/Ethnicity 

Economically Disadvantaged 

Students with Disabilities 

Students with Eimited English Proficiency 

Data for required subgroups have been collected by USOE for several years and is 
generated from the Data Warehouse. Utah school performance and EEA performance 
results are disaggregated by subgroups. 
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5.2 Accountability system holds schools and LEAs accountable for the progress of 
student subgroups. 

Utah’s accountability plan uses three criteria - participation rate, academic achievement, 
and additional indicator - for determining in student subgroups, schools, and LEAs have 
made adequate yearly progress. The Data Warehouse serves as the data source for 
making the determination of AYP for student subgroups, schools, and LEAs. As 
described in section 1.4, the Data Warehouse stores data from test files, clearinghouse 
data (including student demographic information), and S3 year-end data containing 
additional student information for determining membership in relevant subgroups. 

Academic Achievement 

In determining AYP for student subgroups, schools, LEAs and the state as a whole, Utah 
compares the percent of students proficient to the annual measurable objective, examines 
participation rates, and examines the additional academic indicator. 

A student subgroup, school, or LEA of 10 or more students must meet or exceed the 
annual measurable objective for reading/language arts and mathematics, must meet the 
participation rate for subgroups of 40 students or more, and meet the state’s requirement 
for another academic indicator (attendance for elementary and middle schools and 
graduation for high schools). If a student subgroup, school, or LEA fails to meet or 
exceed the annual measurable objective, it must have reduced the percent of students not 
proficient by the appropriate percentage (safe harbor). If the safe harbor provision is 
employed, the additional indicator must be disaggregated and used for determining AYP 
for the subgroup. 

Table 3 displays the impact data for various N sizes on student subgroups. These data 
indicate that the minimum N selected by Utah, N=10, will allow the maximum number of 
schools to be held accountable for student subgroup performance. 



Table 3: Impact Data for Various Minimum N Sizes - Grades 3-8 



Number of Schools with Subgroups Included in AYP Analysis for Various Minimum N Sizes 

(School with >10 students = 522) 




Minimum N 




10 


25 


30 


50 


75 


School 


522 










White 


511 


486 


483 


471 


440 


Low Income 


491 


444 


427 


355 


265 


Students with Disabilities 


450 


280 


227 


63 


11 


Limited English Proficient 


308 


173 


149 


81 


41 


Hispanic 


294 


161 


144 


87 


48 


Black 


29 


2 


1 


0 


0 



In calculating AYP for LEAs, schools, and subgroups, Utah employs a test of statistical 
significance with a one-tailed alpha of 0.01 for academic achievement of the annual 
measurable objective. This allows schools with small subgroup populations to be held 
accountable without falsely identifying a school. This creates a balance between validity 
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(holding schools accountable for all students) and reliability (assuring that those 
subgroups identified have not been so identified simply on the basis of random 
fluctuation of scores). For AYP determination based on the annual measurable objective, 
a test of statistical significance will be applied for subgroups 3 10. The null hypothesis is 
that the observed percent of students proficient in any subgroup is equal to the required 
percent proficient defined by the annual measurable objective. The test of statistical 
significance is a z-score with the distribution of school mean scores (in terms of percent 
correct) around the null hypothesis. A school or LEA makes AYP if the null hypothesis is 
not rejected. 

In calculating AYP, any LEA, school, or student subgroup that does not meet the annual 
measurable AYP goal, must decrease the percentage of students not proficient by at least 
10 percent in accordance with the safe harbor provision. Utah employs a test of 
statistical significance using a one-tailed alpha of 0.25 for determining school-wise safe 
harbor (improvement). 

Participation Rate 

Participation in Core CRTs will be determined for each student subgroup, school, and 
LEA by comparing the testing pre-print file, test answer documents, all student file and 
clearinghouse data using the Data Warehouse. This process is used for all student 
subgroups, schools, and LEAs. Eor details about participation rate, see section 10. 

Additional Indicators 

The additional indicators for Utah’s accountability plan are attendance 
(elementary /middle schools) and graduation (high schools). These indicators can be 
calculated using data in the Data Warehouse. Schools and LEAs must meet the state 
goals for each indicator to make AYP. In addition, a student subgroup must meet the 
additional indicator goals if the safe harbor provision is employed for the student 
subgroup. The state goals for the additional indicators are 93% attendance and 85.7% 
graduation rate. Eor details about these indicators, see section 7. 

Because Utah cannot currently determine graduation rate for all subgroups, a proxy 
additional indicator will be used to determine AYP for 2003-2007. This proxy will be 
attendance. The attendance rate will be obtained by dividing the sum of days in 
attendance by the sum of days in membership across all students in the school and LEA. 
The goal for attendance will be set at 93%, consistent with the additional indicator goal 
for elementary and middle schools (see section 7.2). Schools must meet this goal or 
improve from the previous year. 

Eor a school that meets the annual measurable objective for all subgroups and thus uses 
only school level data for the additional indicator, graduation will be used. However, if a 
school employs safe harbor for a subgroup, AYP will be determined using graduation for 
the total school and attendance for the student subgroup(s). This temporary situation will 
apply equal criteria to all student subgroups while still holding the school accountable for 
graduation. Beginning in 2007, graduation will be the only additional indicator used at 
the high school level. 
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5.3 Accountability system includes students with disabilities. 

For the past four years, Utah has designated two types of alternate assessments for 
students with the most signifieant eognitive disabilities. The purpose of these alternate 
assessments is to measure the aehievement of students with disabilities against alternate 
aeademie aehievement standards as defined by the State. The two assessment approaehes 
provided in Utah are out-of-level testing and Utah’s Alternate Assessment (UAA). 

The Utah Alternate Assessment (UAA) is designed for assessing students with severe 
disabilities, espeeially those with the most significant cognitive disabilities. Objeetives 
for the UAA are based on the student’s lEP goals. By the 2004-05 sehool year, 
alignment of the UAA objeetives with the extended eore eurrieulum will be finalized. 
Profieieney on the Utah Alternate Assessment will be limited to 1% of the total distriet 
enrollment. 

Out-of-level tests are allowed in language arts and mathematics, but only for students 
enrolled in special education. lEP teams may specify the administration of a test that is 
below the grade of enrollment at a level eorresponding with the student’s grade level of 
instruetion. Results are reported in relation to the test level rather than the grade of 
enrollment. In determining AYP, out-of-level tests are eounted as “not profieient”. 
However, out-of-level tests are eounted within the 95% participation. 

The state’s preprinting of student IDs on answer sheets, the seanning and seoring of Core 
CRTs within the state, and the USOE Data Warehouse provide a series of eheeks to 
verify the inelusion of students with disabilities in testing and in aeeountability. 



27 




5.4 Accountability system includes limited English profieient students. 

Limited English profieient students are eurrently ineluded in statewide testing and the 
reporting of results and eomply with NCLB requirements to ensure LEP students 
partieipate in testing. Students with limited English profieieney reeeive 
aeeommodations for testing, as outlined in Utah’s Requirements for Testing 
doeument. These aeeommodations may inelude, but are not limited to linguistie 
aeeommodations and test administration in small group settings. 

Beginning in the 2003-04 sehool year, Utah allowed students who had reaehed 
profieieney and eompleted two years of monitoring, in aeeordanee to Utah’s LEP 
profieieney definition, to be ineluded in the LEP subgroup for AYP reports. 

Students in the United States for less than one year will not be required to take the 
English Language Arts CRT or the Seienee CRT, but will be required to take the 
math CRT. CRT seores for ELL students in their first year in the United States will 
not be used for AYP ealeulations. All other ELL students will partieipate in 
aeeordanee with NCLB with appropriate aeeommodations. These aeeommodations 
may inelude, but are not limited to, linguistie aeeommodations and small group 
settings 

The state’s preprinting of student IDs on answer sheets, the seanning and seoring of 
Core CRTs within the state, and the USOE Data Warehouse provide a series of 
ehecks to verify the inelusion of students with limited English profieieney in testing 
and in aeeountability. 
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5.5 The state has determined the minimum number of students sufficient to yield 
statistically reliable information for each purpose for which disaggregated data 
are used. 

In calculating AYP for subgroups based on the annual measurable objective, and safe 
harbor prowision, Utah employs a test of statistical significance to subgroups 310 for 
each year (Year Two N 320, with no less than 10 in a single year, etc). This allows 
schools with small subgroup populations to be held accountable without falsely 
identifying a school. This also creates a balance between validity (holding schools 
accountable for all students) and reliability (assuring that those subgroups identified have 
not been so identified simply on the basis of random fluctuation of scores). 

For reporting purposes Utah applies a minimum size of ten for all subgroups. 

For the purpose of determining participation rates, as a component of AYP calculation, 
Utah uses a minimum size of 40 for each subgroup. The NCLB requirement for 
participation allows little room for extenuating circumstances when a small group of 
students are involved. This would allow only two students in a subgroup of 40 to not 
participate in testing due to circumstances beyond the control of schools. Utah will, 
however, implement the new flexibility allowed for multi-year averaging of participation. 
For any subgroup, school, or LEA that does not achieve 85% or greater participation in 
any one year, they will not meet the criteria as an average of 95% is not attainable. 



Table 1 1 provides impact data for the number of schools that would have sufficient 
students to include in AYP calculations for various N sizes. This chart clearly indicates 
that many schools would not be accountable for subgroups if an N size greater than 10 
were used. 



Table 11; Impact Data for Various Minimum N Sizes - Grades 3-8 



Number of Schools with Subgroups Included in AYP Analysis for Various Minimum N Sizes 

(Schools with >10 students = 522) 




Minimum N 




10 


25 


30 


50 


75 


Schools 


522 










White 


511 


486 


483 


471 


440 


Low Income 


491 


444 


427 


355 


265 


Students with Disabilities 


450 


280 


227 


63 


11 


Limited English Proficient 


308 


173 


149 


81 


41 


Hispanic 


294 


161 


144 


87 


48 


Black 


29 


2 


1 


0 


0 
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5.6 The State has strategies to protect the privacy of individual students in 

reporting achievement results and in determining whether schools and LEAs 
are making adequate yearly progress on the basis of disaggregated subgroups. 

To assure privacy for students, Utah does not report overall and disaggregated results for 
groups of less than ten students. Requiring ten or more for reporting is acceptable for the 
Family Educational Rights and Privacy Act (FERPA) requirements. 

Test results for student subgroups and schools with less than ten students are not 
displayed either in the hard copy or electronic formats of the school and EEA report card. 
Instead of values, “<10” is displayed. If all or none of students in a student subgroup are 
in the same proficiency level, reports are masked using “<5%” or “>95%” . 
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PRINCIPLE 6: Academic Assessments 



6.1 Accountability system is based primarily on academic assessments. 

Determination of AYP is based on percent of students proficient on Core CRTs, a 
required additional academic indicator (attendance in elementary/middle schools, 
graduation rate in high schools), participation rates, and employment of the safe harbor 
provision. Each of these components is determined for both reading/language arts and 
mathematics. 

Reading/Language Arts: 

Utah uses the Core CRT results from reading/language arts in grades 3-8 and 
grade 10 to determine percent of students proficient for purposes of AYP. 
Proficient is defined as students reaching Level 3 or Level 4. Students in Utah 
enroll in grade specific language arts courses and therefore participate in the grade 
specific test. The Utah CRT Language Arts test has several subtests, all of which 
are included in determining proficiency. The subtests for Grades 3-6 are oral 
language, comprehension, vocabulary, writing, phonics and spelling, and viewing. 
The subtests for Grades 7, 8, and 10 are reading, writing, listening, and viewing. 

Mathematics: 

Students in Utah enroll in grade specific mathematics courses in grades 3-5 and 
participate in grade specific tests. However, at grades 6, 7, 8, and high school 
Utah does not have a single grade-level assessment for use in determining AYP 
because students are allowed to select from a variety of mathematics courses at 
these levels to encourage advanced math achievement. Specifically, students in 6* 
grade may select to enroll in grade 6 math (grade-level course) or pre-algebra, 
students in the 7th grade may select to enroll in Math 7 (grade-level course), pre- 
algebra, or algebra. At the 8th grade, students may advance to the next course in 
the series, instead of all students being enrolled in pre-algebra (grade-level 
course). By high school (grades 10-12), the courses in which Utah students are 
enrolled are quite varied; consequently, Utah does not have a single, grade-level 
test at the high school level 

Utah uses mathematics tests in grades 3-5 that are grade specific. Lor grades 6-8 
the proficiency level for any course in which students are enrolled, at the grade 
level course or higher, is used and reported. This under-represents overall 
mathematics achievement since more advanced students’ scores will not receive 
“additional” credit. In grades 6, 7, and 8 virtually all students are enrolled in a 
mathematics course. Mathematics achievement in grades 10-12 is determined 
through performance on algebra or geometry Core CRTS. 

In compliance with Utah’s current agreement with the Department of Education, 
starting points based on 2002 data and AYP determination for 2003 is based on 
algebra and geometry scores for students. Grades 10-12, enrolled in these courses 
at the high school. 
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PRINCIPLE 7: ADDITIONAL INDICATORS 



7.1 Accountability system includes graduation rate for high schools. 

NCLB regulation 200.19(a)(1) requires use of a graduation rate as the "other 
academic indicator" in determining the AYP of high schools and defines the 
graduation rate for this purpose as "the percentage of students, measured from the 
beginning of high school [presumably 9th grade], who graduate from high school 
with a regular diploma ... in the standard number of years." 

This definition implies a cohort rate. Utah is exercising the option under 
200.19(a)(1)(b) to adjust this definition slightly in order to ensure statewide 
comparability and reduce error in measurement by restricting the cohort to grades ten 
through twelve, since Utah high schools, which may implement any of three different 
grade spans, have only these three grades in common; in fact, nearly half have only 
these three grades - Grades 7-12 (21% of high schools). Grades 9-12 (33%), and 
Grades 10-12 (46%). 

Specifically, the cohort graduation rate is operationalized by Utah, according to the 
recommendation of the NCES (U.S. Dept, of Ed., August 2002, p. 3), to simulate the 
movement of a class through high school: 

The number of students who graduated from 12'*^ grade in the current year 
divided by the sum of: (1) these same graduates, and (2) the number of students 
who dropped out of 12‘^ grade in the current year, (3) 1 1* grade in the prior year, 
and (4) 10* grade in the year before that. 

In order to continue applying official NCES definitions (U.S. Dept, of Education, 
January 2003, pp. 25, 79-81) - in distinguishing "graduates" from "other 
completers", and "dropouts" from "transfers" [see Note below] — which have already 
been incorporated into Utah State Board of Education rule (R277-419), Utah also lags 
the rate by one year; thus, the 2003 report includes the rate for the 2002 cohort. 

Regular diploma graduates may include students with disabilities who can be retained 
as "seniors" until the age of 22. As long as such students are retained, their cohort 
year is adjusted, so that their completion status is included in the calculation of the 
graduation rate for the graduating class of the year in which it is finally determined. 

Since graduation counts are derived from student level detail, they can be 
disaggregated by all of the programmatic and demographic categories identified in 
NCEB legislation. Dropout counts, however, are only available in disaggregated form 
by gender and ethnicity, as required by the NCES in its Common Core of Data 
(CCD) surveys, so graduation rates cannot be calculated for all subgroups. 

To address the discrepancy between NCES/CCD and NCEB/AYP in disaggregation 
requirements, Utah will begin collecting dropout counts at the student level during 
the 2004-05 school year, when the Class of 2007 is in 10th grade. 

In the meantime, for the reporting years 2003-2007, we will use the attendance rate 
(see Section 7.2) as a proxy additional indicator in determining AYP for high schools 
as follows: 
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(a) For a school that meets the annual measurable objective for all subgroups and 
thus requires only school level data for the additional indicator, the graduation 
rate will be used; however, if 

(b) a school employs safe harbor for a subgroup, AYP will be determined using 
graduation for the total school and attendance for the student subgroup. 

This temporary situation will apply equal criteria to all student subgroups while still 
holding the school accountable for its overall graduation rate. Beginning with the 
Class of 2007 (in reporting year 2008), a fully disaggregated graduation rate will be 
the only additional indicator used at the high school level. 

NCLB regulation 200.16(a) specifies the 2001-02 school year as the base year for 
establishing the "starting point" for measuring AYP. Utah interprets this to mean that 
we should derive a "cut score" from the graduation rate of the Class of 2002. Using 
available data and method outlined above, we estimated the graduation rate for this 
cohort from the beginning of their 10th grade year through the end of their 12th grade 
year by ethnicity (with the count of regular diploma graduates in parentheses): 

Asian = 84% (502) 

Black = 65% (172) 

Hispanic =65% (1,574) 

Indian =67% (313) 

Pacific Islander =76% (315) 

White = 88% (27,306) 

Total = 86% (30,182) 

The following decision rules will be applied to each unit of analysis (school or LEA): 

• If the cohort contains less than 40 students, there are too few students to make 
a reliable decision about the graduation rate, the graduation rate indicator is 
"not applicable" to the unit. 

• If the cohort is equal to or greater than 40 students and the graduation rate is 
greater than 85.7%, the unit "passes"; 

• If the graduation rate is less than 85.7% but greater than the graduation rate 
for the prior year cohort, the unit "passes"; 

• In all other cases the unit "fails". 

The 85.7 percent cut score is just below the rate for the entire cohort in the base year 
(see the Total row. Table 12). As the table shows, this will be an ambitious goal for 
several of the ethnic groups and presumably for the students with disabilities and 
limited English proficiency populations. 

The figure of 85.7 has a meaningful interpretation in terms of the event dropout rate. 

It is obtained by accepting a maximum event dropout rate of 5 percent per year over 
the life of the cohort. Thus, 95 percent of the cohort remains at the end of 10th grade, 
90.025 percent (95% of 95%) remains at the end of 1 1th grade, and 85.738 percent or 
(rounded) 85.7 percent (95% of 95% of 95%) remains at the end of 12th grade, the 
normative time of graduation. 
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Tablel2: Dropout Event Rate by Grade 



Grade 


2001 Fall Enrollment 


2001-2002 Dropout 
Event Count 


Dropout Event Rate 


7 


35,538 


367 


1.03% 


8 


35,786 


396 


1.11% 


9 


35,029 


519 


1.48% 


10 


36,118 


843 


2.33% 


11 


35,923 


1,387 


3.86% 


12 


34,951 


2,430 


6.95% 



Finally, note that this figure is not a starting point as sueh, but a eonstant applied 
every year, since under NCLB regulation 200.19(d)(1), "states are not required to 
increase the goals of its others academic indicators over the course of the timeline." 
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7.2 Accountability system includes an additional academic indicator for elementary 
and middle schools. 

NCLB regulation 200.19(a)(2) requires "at least one academic indicator" for 
elementary and middle schools but does not identify a specific indicator. In 
subsection (b)(3) "attendance rates" are mentioned as a possibility, and Utah uses the 
attendance rate for this purpose. 

Unfortunately, Utah cannot empirically derive a cut score from the 2001-02 school 
year for the attendance rate as done with the graduation rate, because attendance data 
were not collected at the state level in that year nor for several years — funds are 
allocated to districts on the basis of membership rather than attendance, and the 
NCES/CCD has not required reporting attendance data since 1992. Utah, therefore, 
relies on a recommendation from the Center for Assessment (NCIEA, 2003), which 
has considerable experience in consulting on the construction of educational 
accountability systems in several states, as to what constitutes a reasonable value for 
this purpose. 

Accordingly, units of analysis that achieve or exceed an attendance rate of 93 percent, 
as well as those that are below that standard but have improved their attendance rate 
when compared with the prior year, are considered as having met the additional 
indicator requirement of AYP. 

The attendance rate is obtained by dividing the sum of days in attendance by the sum 
of days in membership across all students in the unit of analysis. Because attendance 
data is collected by the USOE at the student level, it can be disaggregated by all 
student subgroups when a school employs the safe harbor provision. 
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7.3 Additional indicators are valid and reliable. 

Regarding the graduation rate, the definitions of "graduate" and "dropout" used by 
each LEA and the formula used by the USOE to calculate the rate explicitly comply 
with NCES standards. In addition, an independent accounting firm formally audits 
the dropout count in each EEA annually following these guidelines (State of Utah, 
APP-C5): 

"2a. Prom the prior year attendance records, select a sample of students who did 
not complete the prior school year in grades 7-12 and were NOT identified by the 
district as dropouts. The sample size should be equal to or greater than 5% of the 
actual number of such students or, in other words, of the sampling frame; 
however, the sample size should not be fewer than 10 and need not exceed 30. 

"2b. Por the sample of students defined in 2a, determine whether the students 
have been properly excluded from the dropout totals as reported in the district’s 
Annual Public School Pall Enrollment Report in accordance with SOE’s rules for 
pupil accounting R277-419-5. In other words, determine whether adequate 
documentation exists to support the district's claim that the student was not a 
dropout." 

Regarding the attendance rate, each EEA uses the same definition: 

"Attendance" is the total number of days a student attended a specific school. A 
student is counted as "in attendance" on a school day if the student was counted 
on the class role by a teacher as being "present" — in Grade 1-6, at any time 
during the day; and in Grades 7-12, in at least one period of the day. 

The formula used for calculating the attendance rate reflects actual student behavior 
in the aggregate, on the one hand, and the school's differential responsibility for each 
student, on the other: a student enrolled for a shorter period with perfect attendance 
appropriately enhances the school's performance, but a student enrolled for a longer 
period with poor attendance is appropriately given more weight by virtue of having 
more days in membership, so the school has an incentive to improve that student's 
attendance. 

NOTE 

Por the purposes of NCEB/AYP, the completion status of each secondary (Grade 7- 
12) student who leaves a Utah public school is (or, when dropout data collection is 
fully automated, will be) represented in the Data Warehouse by an "exit code" 
according to the following classification: 

• Graduate (numerator and denominator) = "GR" or any of six "early 
graduation" codes. 

• Dropout (denominator only) = "DO" or "CT" (the latter is a student who 
"completes" school without obtaining a "regular diploma"). 

• Transfer (excluded from calculation) = any of seven codes identifying 
legitimate ways of leaving school, including death ("DP") and withdrawal 
("WD") due to illness or accident. 
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Students who have been "expelled" ("EX") or are suspended ("SU") at the end of the 
year are also treated as dropouts, unless their status ehanges by the beginning of the 
following year. 
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PRINCIPLE 8: SEPARATE DECISIONS EOR READING/EANGUAGE ARTS AND 
MATHEMATICS 



8.1 Accountability system holds schools and districts separately accountable for 
reading/language arts and mathematics. 

The AYP calculation examines separately the percent of students proficient in 
reading/language arts and mathematics. In addition, participation rates are examined for 
each subject separately. Baseline values and annual measurable objectives are defined for 
each subject as well. AYP is determined for each student subgroup, school, EEA and the 
state as a whole for reading/language arts and mathematics. 
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PRINCIPLE 9: SYSTEM VALIDITY AND RELIABILITY 



9.1 Accountability system produces reliable decisions. 

Utah has incorporated several features into the aeeountability program to ensure reliable 
deeisions. 

• Core CRTs used as aeademic measures have been developed using industry 
standards to produce valid and reliable seores. Use of these tests to make AYP 
decisions sets a foundation for making reliable deeisions. 

• Use of statistical significance tests instead of an absolute minimum N for 
annual measurable objeetives will reduee the probability of a Type 1 error 
(falsely identifying a school for program improvement). 

• An LEA is identified for improvement only when it misses AYP in the same 
subjeet and in all grade spans for two eonseeutive years, or the other aeademic 
indicator in all grade spans for two consecutive years. 

• Use of statistical significance tests and multiple years of data for making safe 
harbor deeisions will reduce the probability of Type 1 errors. 
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9.2 Accountability system produces valid deeisions. 

Utah ineorporates several key proeesses to ensure valid deeisions about AYP. 

• Tests of statistieal signifieanee allow all sehools to be aeeountable for all 
student subgroups regardless of group size. 

• All students are ineluded in Utah’s aeeountability plan, and monitoring 
proeesses are in plaee to verify inelusion. 

• Appeals will be allowed at the sehool and LEA level. A sehool may have 30 
days to appeal an AYP deeision to the LEA. An LEA may have 30 days to 
appeal an AYP deeision to USOE. 

• USOE has a validation of data proeessing proeedure in plaee eonsistent with 
the established timeline for data submission, proeessing, and return of results. 

Sehools may appeal the proposed identifieation of not making AYP for statistieal or other 
substantive reasons, sueh as eatastrophie events that may have eaused errors in test 
results. The proeess for teehnieal appeals is detailed here. 

Quality eontrol and a ehanee to eorreet any errors are eritieal to the proeess of ereating a 
valid and reliable system. Student data (baekground & behavior) and student 
performance in the aeademie areas of math and language arts serve as foundations for 
determining AYP for student subgroups, sehools, and LEAs. These student level data are 
submitted at eertain intervals during the sehool year via the USOE elearinghouse 
( http://desntl.usoe.kl2.ut.us/Clearinghouse/Clearinghouse.htm) . These data are 
validated through quality assuranee proeedures, summarized and forwarded to the USOE 
data warehouse (http://www.usoe.kl2.ut.us/upass/) . AYP analysis will be performed on 
the needed USOE warehouse data elements. 

Eormal and informal appeals proeesses oecur at numerous steps along the way. The 
following are key data submissions, their timeline and how eaeh element and the overall 
sehool status may be appealed. 

Student Pre-Print Data (March/April) - Preprint student data is submitted to USOE. 
This data ineludes student demographie data. At the distriet and sehool level, this data is 
summarized and sent to distriets to approve or make eorreetions in their operational 
student data and then resubmit it. This individual information is used to pre-print Core 
CRT answer sheets. This proeess both reduees work for the students and improves 
student data. Distriets provide written approval of their pre-print data before answer 
sheets are printed. 

Validation and Appeals: Distriets ean make eorreetions and resubmit data up to testing 
window. 

Core CRT Scanning & Scoring (May/June) - Math and Language Arts Core CRTs are 
seanned and seored by USOE. Utah is one of only a few states that do in-house seanning 
and seoring. This approaeh leads to faster seoring turnaround, stronger quality assuranee 
proeedures and more aeeurate test data transfer to our USOE Data Warehouse. 
Throughout the seanning proeess, ineonsisteneies and problems with student answer 
sheets are identified and eorreeted. Depending on the issue, either the distriet or sehool is 
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directly involved. USOE scoring staff is assigned certain districts. This has enabled a 
relationship to develop with the districts and increases the cooperation as scanning or 
scoring issues arise. Scoring turnaround and testing procedures are well documented in 
Utah Board rule, R277-473-3. Time Periods for Administering and Returning Test 
Materials. 



Validation and Appeals: Districts and schools work directly with USOE scoring staff to 
resolve any inconsistencies or possible errors. These issues and their resolution are 
documented and dated by USOE staff. Basic score distributions are also examined to 
assure students are being scored against the correct answer key. Districts must signoff on 
preliminary raw score results in June. In addition, USOE Scoring staff compare raw score 
results at the school and grade level with previous years’ results to check for testing 
irregularities and assure quality data. 

Clearinghouse “End of Year” data submission (July) - Erom an accountability 
perspective, this student level data submission provides background characteristics, grade 
level data, enrollment date, exit date, graduation status, and attendance. This serves as the 
foundation of student level identification to which test data is matched. These data are 
passed on to the USOE Data Warehouse for integration with the other data systems. 

Validation and Appeals: Districts are encouraged to submit this large data file at the start 
of June. Once data are submitted, USOE analysts verify the structure and format of the 
data. An edit check program is run against the data to verify its quality and completeness. 
Written reports and diagnostics are sent to districts regarding this data. Districts are to 
make necessary corrections and modifications. Einal clearinghouse data are due July 15. 
By that point, districts must provide written verification that data are accurate and 
complete. 

Teacher and Course Data (i.e., CACTUS) submission (July) - Teacher and course 
level data is maintained in the CACTUS database. This database is centralized by USOE. 
CACTUS serves as the operational teacher system and is updated throughout the school 
year. Districts are required to have all changes into the system finalized by July 

Validation and Appeals: USOE staff work on a case-by-case basis with districts. 

Regular training on CACTUS and updates to the data system are conducted during the 
year. Districts receive written notification reminding them of the CACTUS data 
submission deadline. The role of schools and districts is to assure the data is accurate and 
up to date by the end of the school year. 

Behavior and Assessment Data Merge (July/August) - USOE data warehouse merges 
various data systems needed for AYP calculations. Utah has common naming and id 
system that districts should have employed. 

Validation and Appeals: USOE data warehouse staff will provide districts and their 
schools data on the success rate and issues regarding matching and merging of student 
data. Depending on the nature and extent of the problem and overall timelines, districts 
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may resubmit a data element. The State Associate Superintendent in charge of data will 
ultimately make this decision. 

Appeals and Consequences of missing data deadlines - Board Rule R277-484 . Data 
Standards, Deadlines and Procedures gives clear and specific detail regarding data 
deadlines. Districts must meet data deadlines or funds will be withheld. This rule also 
specifies procedures to get an extension for a deadline. 

District and School review of AYP status - Utah schools and districts will have 30 days 
to appeal the AYP decision as long as they meet all of their data submission 
requirements. All data elements (outlined above), statistical procedures, merging of 
student data (outline above) and AYP status will be reviewed. Appeals must be 
submitted in writing between schools and the LEA and between the LEAs and the SEA. 
The SEA and the EEA will follow all necessary procedures detailed above. The State 
Superintendent and the Utah State Board of Education will make final decisions 
regarding appeals and missing data deadlines. 
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9.3 State has a plan for addressing changes in assessment and student population. 

Core CRTs are post-equated using established psychometric methods and procedures. As 
new or revised assessments are administered, they will be included in the AYP 
determination. 

The baseline for newly opened schools and schools experiencing significant changes in 
student population due to boundary changes will be based upon the LEAs most recent 
data. Charter school baseline data will be based on the district in which it is located. 

These data will be compared to the school’s actual data generated during its first year of 
operation for AYP calculations. In subsequent years, data generated by the school will be 
used for making AYP determinations. The goal of 100 percent proficiency by 2013-2014 
will be established for new schools. 
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PRINCIPLE 10: PARTICIPATION RATE 



lO.I Accountability system has the means for calculating the rate of participation in 
the statewide assessment. 

Utah has several proeesses in place to account for all students and make a determination 
of participation rate. 

• Answer documents are generated by the pre-print file and submitted to the 
USOE by each LEA Each Core CRT answer sheet includes a special codes 
box that is completed for each student. Participation and non-participation are 
indicated in the special codes box. All answer documents are returned 
regardless of student participation. 

• The all-students files submitted by LEAs are used as part of the scoring 
system. Student records are matched with test results. 

• The final participation rate is based on the number of students tested 
eompared with the number of students enrolled at the end of the test 
administration window. 

The 95 percent participation rate will apply to student subgroups and schools with a 
minimum size of 40. Utah will, however, implement the new flexibility allowed for 
multi-year averaging of participation. Eor any subgroup, school, or LEA that does not 
achieve 85% or greater participation in any one year, they will not meet the criteria as an 
average of 95% is not attainable. 

The 95 % participation rate will be based on membership at the school or LEA. All 
students enrolled at a school must participate in testing whether or not the student’s score 
will be part of the academic achievement determination as part of AYP. Students not 
included in the school academic achievement determination may, in fact, be eounted at 
the LEA level, necessitating a test score. 
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10.2 Accountability system has a means of applying the 95 pereent assessment 
eriteria to student subgroups and small sehools. 

Schools buildings and LEAs will be required to administer Core CRTs to all students 
enrolled at the time of test administration. Sehools and LEAs that inelude 95 pereent of 
the enrolled students meet this requirement for AYP. Partieipation is ealeulated for 
reading/language arts and mathematies separately. Those sehools and LEAs in which less 
than 95 percent of any student subgroup participates in testing do not meet the AYP 
standard, provided that the subgroup size is at least 40. 
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